Limit recursive sketch compilation to the src directory
#148
Conversation
This is a new helper function that takes care of listing files with given extensions in a directory, recursively or non-recursively. This functionality was previously duplicated in a few places. This new helper function now replaces some duplicate code in SketchLoader and CollectAllSourcesFilesFromFoldersWithSources. Compilation of code in the core and libraries still uses two other mechanisms for listing files (in builder_utils), which will later be converted to use this new helper function too. This new helper function has some subtle differences compared to the previous code: - The new code raises an error when the root directory does not exist, which the CollectAllSourcesFilesFromFoldersWithSources code did not do. To compensate for this, LibraryToSourceFolder will only return the "utility" folder when it exists. - The new code completely ignores any hidden or source control folders or files. Originally, arduino-builder showed a warning when source control or hidden folders were present, and then compiled files inside of them anyway. Since f5edd23 (Avoid warning about SCCS folders in libraries, just ignore them) this warning is no longer shown in most cases, but the files themselves are still included in compilation. With this commit, they are just completely ignored instead (for sketch compilation and include detection, for core and library compilation they will are still used). See also arduino#139. Signed-off-by: Matthijs Kooijman <[email protected]>
Since c7a18f8 (Sketch sources files were recursively copied BUT not compiled), sketches were compiled recursively. However, this turned out to have some side effects, such as including files that were not intended to be compiled (included as documentation or otherwise not part uof the sketch itself) and breaking the build if the build directory is inside the sketch directory (arduino#89). To prevent these side effects, but still allow users to apply more structure to their sketches, recursive compilation is now limited to the `src` directory inside the sketch folder. All .ino files must still reside in the root folder, but any .c and .cpp files inside (subdirectories of) the `src` directory are included in the build. Using a `src` directory in this way is similar to how libraries are compiled (except for libraries *all* source files live in the `src` directory if it is present, while for sketches files can *also* live in the root directory). Sketches written to use this new `src` directory should also be backward compatible, in the sense that arduino-builder 1.0.7 (Arduino IDE 1.6.7) and later can succesfully compile it, not just the versions after this commit. The reverse is not true: sketches that need recursive compilation outside of the `src` directory, as supported by arduino-builder 1.0.7 above, will not compile after this commit, but it is trivial to convert them to do so (without breaking compilation before this commit). This also changes some testcases that used subfolders, to move these inside the `src` directory instead of directly in the sketch root directory. This fixes arduino#89. Signed-off-by: Matthijs Kooijman <[email protected]>
|
✅ Build completed. ⬇️ Build URL: ℹ️ To test this build:
|
|
So is this going to fixed or not? @matthijskooijman There is no need for the build system to have this restriction to try enforce a set of artificial limitations, especially when it is enforcing limitations that have not existed in the past. This is very similar to the restrictions that were placed on new 1.5x libraries in the IDE releases 1.5 through 1.5.5 where a library was totally ignored if the code wasn't down under a directory named "src". This choice to not allow sub directories in a sketch folder seems like a similar decision.
|
|
I posted a suggested very simple fix in the other issue here: |
I agree this is a very large change in behavior for a minor revision. To follow semantic versioning, shouldn't this have bumped the IDE to 2.0.0, not 1.6.10 -- like what happened when the API changed from 0023 - 1.0.0. The purpose for this type of versioning is to let people know there may be things missing or at the least, breaking changes which they should expect. A bump from 1.6.9 to 1.6.10 should only tell me there are either improvements, new features, and bug fixes. |
|
This may be stating the overly obvious, but the Arduino IDE as a complete system doesn't really support sources in sketch subdirs as long as the IDE's editor doesn't support at least showing the files to users. I have personally helped at least a couple end users who ended up with an extremely confusing (and very tough to diagnose remotely) situation where somehow a subdir got created with more files having conflicting copies of their variables. When the extra files are all but invisible to an end user, it's a recipe for terrible confusion. On these github issues, it's easy to focus on technical details. Just want to post a gentle reminder that whatever is done in the builder needs to be carefully considered in how it's presented to use users by the GUI. |
See arduino/Arduino#5186 Partially revert #148 Signed-off-by: Cristian Maglie <[email protected]>
- shortened serial UI commands to save bytes - adapted folder structure to new limitation of arduino 1.6.10 whereby only source files in the ‘src’ subfolder of a sketch are being included => moved “subprojects” into a new ‘src’ folder (config, log, sensors, ble_gatt). For details see arduino/arduino-builder#148 - arduino 1.6.10 uses the latest AVR tooling => boiler controller with BLE uses only 25.5 KB now instead of 26.9 KB
|
I am another victim of this change. I have lost eight hours messing with my code bases trying to figure out why nothing would compile("undefined reference to") on one computer and why it would not find any files at all on another computer.(I upgraded from Arduino IDE 1.6.5 to 1.8.2.) This really should have been a major revision changed to 2.x.x as @Chris--A mentioned or at least have system built in place to warn users when they #include from non-whitelisted directories. As it stands now the Arduino documentation does not mention any limitations to include folder structure and the error messages thrown are cryptic when the file paths appear to be typed correctly. |
|
I also spent a good time googling around today. It's very awkward from programmer perspective, any other compiler never had a limitation like that. |
|
Is this ever going to be fixed? This is a ridiculously arbitrary limitation. |
|
It's really sucks. Wasted a whole night because of this feature. |
|
I doubt they will ever go back and fix the versioning to reflect this major change. So I doubt anything else will happen due to the lack of developer comment follow up. |
It is ironic that the library manager back-end scripts require that Arduino libraries use semver compliant version numbers/strings and in their github git tags, yet the IDE totally ignores the point of semver versioning. IMO, the issue with the src directory and sub directory build issues is that the IDE made some wrong-headed decisions quite a while back. One of those was to not compile with the sources left in place. |
|
The restriction of recursive compilation to the Indiscriminate recursive compilation was done from Arduino IDE 1.6.6 to 1.6.9 but this feature was never advertised so I doubt many people discovered and took advantage of the new feature over that time and then found their code to be broken by the restriction of recursive compilation to the I think restricting recursive compilation to the The feature does really need to be documented so that more people will be aware of this very useful feature. I would like to see an Arduino Sketch Specification page created, similar to the existing Arduino Library Specification page. |
@per1234 , maybe you (and all of us) could start to put some pressure on the Arduino dev team to actually follow it with their s/w releases, especially for things like the IDE package? I think part of the issue is that for years everything is called "Arduino" , From the board, to the "language" to the "tool" use to build the sketches. It can make it difficult when trying to refer to various sub components - which might also have their own versioning. I download and used Arduino to upload a sketch written in Arduino, to my Arduino.... And then there are several sub-components within the Arduino IDE. But at least IMO, I agree with @Alexia that since the IDE that is download has its own version number, that the version number for the Arduino build package/IDE should be a semver version number and follow semver versioning. As it is now, the IDE version number does not seem to follow semver semantics at all. --- bill |
I am one of the people for whom this change (restriction to
I take issue with it because I believe it to be a surprising restriction contrary to common practice and expectations. Furthermore I take issue with it because it forces people to have one of their folders follow a particular arbitrary naming convention (i.e. the use of contractions) that may otherwise look out of place in the rest of their code. I would also be interested to know why people were attempting to have more than one program per 'sketch'. |
…andard Arduino libraries. Rather than requiring these libraries to be manually relocated to the Arduino "libraries" directory, just refer to them in place. Note for an Arduino project to support this file structure all sub-directory includes must be contained within a "src" sub-directory. The sub-directory limitation was introduced in 1.6.10, see arduino/Arduino#5176, and arduino/arduino-builder#148
No description provided.